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(57) Abstract: The perceptibility of degradations caused to signals transmitted over a transmission medium is measured by gen- 
erating one or more predetermined transmission degradation conditions (e,) and subjecting a test signal (S.) to the transmission 
degradation conditions (e,) in a network simulation device (3). The degree M(e) to which the or each transmission degradation con. 
dition (e,) imperceptible to the human perceptual system is measured and a data set is generated and stored for converting one or 
more transmission degradation conditions (e t ) to respective values of perceptibility M(e). The data set may be a look-up table or an 
empirically determined formula. The data set may then be used on live traffic (S\), by identifying objectively measured transmis- 
sion degradation conditions (e) in the received signal (S') and retrieving from the data set in the data storage means (7) a value of 
perceptibility (M) associated with the transmission degradation conditions (e) so identified. In this way a measure of the subjective 
significance of degradation in the signaJ (S\) can be derived from objective measures of degradation 
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IN-SERVICE MEASUREMENT OF PERCEIVED SPEECH 
QUALITY BY MEASURING OBJECTIVE ERROR PARAMETRE 



This invention relates to the assessment of the condition of 
telecommunications systems whilst in use. 
5 Signals carried over telecommunications links can undergo considerable 

transformation, such as digitisation, data compression, data reduction, amplification, 
and so on. Further distortions can be caused by electromagnetic interference from 
external sources 

Objective processes for the purpose of measuring the quality of a signal are 

10 currently under development and are of application in prototype testing, pre-delivery 
testing of components, and in-service testing of installed equipment. They are most 
commonly used in telephony, but are also of application in other systems used for 
carrying speech signals, for example public-address systems. 

The present applicant has a number of patents and applications relating to 

15 this technical field, most particularly European Patent 0647375, granted on 14 th 
October 1998, and its foreign equivalents. In this invention, two initially identical 
copies of a test signal are used. A first copy is transmitted over the communications 
system under test. The resulting signal, which may have been degraded by the 
system under test, is compared with a reference copy of the same signal, which has 

20 not passed through the system under test, to identify audible errors in the degraded 
signal. These audible errors are assessed to determine their perceptual significance - 
that is, errors of types which are considered significant by human listeners are given 
greater weight than are those which are not considered so significant. In particular, 
inaudible errors, being perceptually irrelevant and therefore uneccssary for 

25 assessment, are disregarded. 

The automated system provides an output comparable to subjective quality 
measures originally devised for use with human subjects. More specifically, it 
generates two values, Yle and Ylq, equivalent to the "Mean Opinion Scores" (MOS) 
for "listening effort" and "listening quality", which would be given by a panel of 

30 human listeners when listening to the same signal. The use of an automated system 
allows for more consistent assessment than human assessors could achieve, and also 
allows the use of compressed and simplified test sequences, which give spurious 
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results when used with human assessors because such sequences do not convey 
intelligible content. 

Different errors may have different perceptual significance in different 
languages, according to the use of different phonemes, and their relative frequencies 
5 of occurrence, in different languages. Multilingual test sequences are also more 
readily accommodated in automated systems than those using human subjects, as 
the human subjects would need to be familiar with all the languages represented in 
the test signal, again to avoid spurious results caused by any lack of intelligibility to 
the listener. 

10 An auditory transform of each signal is taken, to emulate the response of the 

human auditory system (ear and brain) to sound. The degraded signal is then 
compared with the reference signal in the perceptual domain, in which the subjective 
quality that would be perceived by a listener using the network is determined from 
parameters extracted from the transforms. 
15 Such automated systems require a known (reference) signal to be played 

through a distorting system (the telephone network) to derive a degraded signal, 
which is compared with an undistorted version of the reference signal. Such systems 
are known as "intrusive" measurement systems, because whilst the test is carried 
out the system under test cannot carry live (revenue-earning) traffic. 
20 A suitable test signal is disclosed in European Patent Specification 0705501 

and comprises a sequence of speech-like sounds, selected to be representative of the 
different types of phonetic sounds that the system under test may have to handle, 
presented in a predetermined sequence. The sounds are selected such that typical 
transitions between individual phonetic elements are represented. Typical speech 
25 comprises a sequence of utterances separated by silent periods, as the speaker 
pauses to breathe, or listens to the other party to the conversation. These silent 
periods, and the transitions between utterances and silent periods, are also modelled 
by the test signal. 

The arrangement described above requires the use of a pre-arranged test 
30 sequence, which means it cannot be used on a live telecommunications link - that is, 
a link currently in use for revenue-earning traffic - because the test sequence would 
interfere with the traffic being carried and be audible to the users, and because 
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conversely the live traffic itself (whose content cannot be predetermined) would be 
detected by the test equipment as distortion of the test signal. 

In order to carry out tests on equipment in use, without interfering with the 
revenue-earning traffic being carried by the equipment (so-called non-intrusive 
5 testing), proposals have been made to use the live speech signals themselves as the 
test signals. This can be achieved quite simply if an undegraded copy of the original 
signal is available at the point of measurement. In test conditions it is, of course, 
possible to place the transmitter and receiver in close proximity, but this is impossible 
when testing an in-service long-distance link. In a real situation any reference signal is 

1 0 likely to be subject to similar degradation to that experienced by the link under test, 
unless a significant error correction overhead is provided to ensure that the reference 
copy is itself received undegraded. 

German Patent Specification 4324292 (DeTeCon) of 1995 discloses a 
system in which an empirical relationship is determined between an objective 

1 5 measure (such as bit-error rate) and a subjective measure (that is, one made by 
human subjects), such as Mean Opinion Score. This derived relationship is then used 
to estimate the subjective quality of a received signal from an objective measure 
made of the live network. For this arrangement to produce reliable results sufficient 
repetitions of each subjective test must be made to provide a statistically reliable 

20 value for the Mean Opinion Score. Such a system is not practical for any but the 
simplest of systems. The search space for determining an accurate relationship 
between an objective measure such as packet statistics and a subjective meausure of 
speech quality is very large. To determine a person's opinion of speech quality needs 
several seconds of speech to analyse. However, it can take many days or even 

25 weeks to analyse the objective speech properties and assess them relative to the 
subjective values reported by the human subjects. Should a particular property, or 
range of values of a given property, prove to be of particular interest (perhaps 
because of a large variation in subjective value for a small change in objective value) 
it is rarely practical to repeat the tests with a greatrer definition, as the same human 

30 subjects may not be available or may give less consistent responses (for health or 
other reasons) when the test is repeated, perhaps several weeks after the original 
test. Any complex system comprises a large number of network elements, each 
having a variety of possible behaviours, which will interact with each other. It would 
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not be practicable to perform subjective tests using human subjects for more than a 
small proportion of the possible network behaviours. Furthermore, in a packet 
switching system each individual data packet may be transmitted by a different 
route, using different network elements. Since degradation will in general be different 
5 for each such route, the effect on the complete message will be much more difficult 
to assess. 

The present Applicant's Internationl Patent specifications WO96/06495 and 
WO96/06496 (both published on 29th February 1996), and WO97/05730 (published 
13 th February 1997) propose three other possible solutions to this problem. 
10 WO96/06495 describes the analysis of certain characteristics of speech which are 
talker-independent in order to determine how the signal has been modified by the 
telecommunications link. In WO96/06496, the content of a received signal is 
analysed by a speech recogniser and the results of this analysis are processed by a 
speech synthesiser to regenerate a speech signal having no distortions. 

1 5 W097/05730 discloses a process in which the received signal is compared with 
parameters identifying speech-like and non-speech like characteristics, to derive a 
measurement of the quality of the signal. However, all of these techniques require 
considerable processing power, making it difficult to monitor signal quality in real 
time, and the processes all require some assumptions to be made about the incoming 

20 signal. 

A further proposal, disclosed in the Applicant company's International Patent 
Application PCT/GB0 1 /02050 filed on May 10 th 2001 provides a method using the 
simpler "intrusive" measurement techniques in a non-intrusive manner on a 
communications channel which is carrying a live call, intended for use for voice 

25 communication systems, in which there are normally gaps in the speech content. A 
predetermined test signal is transmitted over the communications link during such 
gaps, and compared at the remote end with a reference copy of the signal. However, 
this system is only suitable for systems in which there are suitable gaps in the 
information content. It is not suitable for systems in which such gaps are used for 

30 other purposes, such as carrying data or other voice calls, or for "half-duplex" 
systems in which both directions of conversation use the same channel. 

According to the present invention, there is provided a method of measuring 
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the perceptibility of degradations caused to signals transmitted over a transmission 
medium comprising the steps of: 
generating a test signal; 

generating one or more predetermined transmission degradation conditions; 
5 subjecting the test signal to the or each transmission degradation condition; 

using an objective measure to determine the degree to which the or each 
transmission degradation condition is perceptible to the human perceptual system; 

generating therefrom a data set for converting transmission degradation 
conditions into values of perceptibility; and 
10 storing the data set so generated. 

An objective measurement system suitable for the purpose is the 
"Perceptual Analysis Measurement System" (PAMS) disclosed in the present 
applicant's European patent 0647375, and available from Psytechnics Ltd of Ipswich, 
UK. 

15 To use the data set so stored, the method may comprise the further steps of: 

receiving a signal from a transmission medium; 

identifying transmission degradation conditions in the received signal; 

retrieving from the stored data set a value of perceptibility associated with 
the transmission degradation condition so identified; 
20 thereby generating a measure of the perceptibility of the degradation in the 

received signal. 

By using an objective system it is possible to automate the calibration 
process, making it simpler and more relaible than the subjective methods used in the 
prior art reference. The transmission degradation conditions may include values of 

25 bit error rate, packet delay variation, packet loss characteristics, or other conditions. 
These can be simulated using a network emulator. 

The data set may be stored in the form of a "look-up table" or preferably as 
an empirical relationship, determined using "best-fit" techniques, between the error 
conditions e and associated objectively-measured estimates of subjective error values 

30 M(e), so that the subjective error M(e) is defined as a mathematical function of the 
measured values of packet loss, jitter etc. 
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In a preferred arrangement values of perceptibility are selected according to 
the nature of elements of the transmission path located before and after the 
transmission medium under test. 

In a further embodiment, the objective speech quality measure is used to 
5 improve the efficiency of search of the test space, by imposing an initial plurality of 
degradation conditions on the system, and then selecting further degradation 
conditions according to the degree of variation in the resulting perceptibility 
measurement value. The system can be used to identify values of objective 
properties for which a predetermined variation in value generates relatively large 
10 differences in the resulting perceptibility measurement values, and performs 
additional measurements of such properties for neighbouring values. Conversely, if 
the speech quality measures are showing little difference between conditions then 
i the the degree of variation can be increased. The system may also perform tests 
between the incremental steps to identify local maxima and minima. Thus, the 
15 immediateness of the objective perceptual speech measure can be used to improve 
calibration efficiency. 

As an example, a test is performed on a specified network condition, and the 
speech quality is then immediately calculated. The network condition parameters are 
then altered by an adjustment factor, and the quality calculated again. This process 
20 continues and if the quality score differs by an amount greater than a defined 
threshold, over a specified number of measurements, the tester reduces the 
adjustment factor. The tester may also retrack through the condition settings which 
showed the large variation with the new adjustment factor. 

Apparatus suitable for performing these methods form further aspects of the 
25 invention. 

The invention analyses the degradation of the actual signal being 
transmitted, rather than of a test signal as in the Applicant company's earlier 
application PCT/GB01/02050 mentioned above. However, the requirement for 
calibration means that this invention will only respond to fault conditions for which it 
30 has been calibrated. 

Apparatus for performing this method is also included in the scope of the 
invention. 
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Embodiments of the invention will now be described, by way of example, 
with reference to the drawings, in which: 

Figure 1 illustrates diagrammatically a system configured to perform the 
calibration process according to the invention; 
5 Figure 2 illustrates diagrammatically a system configured to perform the in- 

service measurement process according to the invention. 

The calibration system illustrated in Figure 1 has a test signal generator 1 , 
which generates a predetermined signal St (such as that disclosed in European Patent 
0705501) for transmission by a transmitter 2 over a network simulator 3 to a 
10 receiver 5. A fault simulator 4 is arranged to control the simulator 3 such that it 
imposes deliberate error conditions et on signals St transmitted through the simulator 
3, such that the receiver 5 receives a degraded signal St'. The error conditions 
simulated are of a type readily measured objectively, such as different values of bit 
error rate, packet delay variation ("jitter"), and packet loss characteristics (number of 

15 packets lost and any pattern to them). However, their subjective significance 
depends on a number of factors, including the type of transmitter and receiver used, 
and is not known at this stage of the process. 

An analysis unit 6 is provided to compare the original signal St with the 
degraded signal St' received by the receiver 5. The analysis unit 6 operates according 

20 to the principles described in the aforementioned European patent 0647375 to 
generate a measure M of the subjective importance of the errors detected. The 
subjective importance depends on a number of factors. Empirical "Mean Opinion 
Scores" given by human subjects to degraded signals show a highly complex 
relationship with the incidence of errors in the signal. 

25 The output M corresponding to each error condition et simulated in the 

network simulator 3 is passed to a store 7 which compiles a set of error conditions e 
and associated subjective error values MfeJ. These values may be stored in the form 
of a "look-up table". Alternatively, an empirical relationship between the error 
conditions e and associated subjective error values MfeJ may be determined using 

30 known techniques, such as neural networks, or radial basis function networks, so 
that the subjective error MfeJ can be defined as a mathematical function of the 
measured values of packet loss, jitter etc, and this function can be stored. The look 
up table of stored values, or subjective error function M(e), can then be used on a 
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real network 13, as is illustrated in Figure 2. The results will be consistent provided 
that the transmitter 2 and receiver 5 used are similar to those (2,5) used in the 
calibration process. 

Unlike the calibration process, the transmitter 2 transmits a live signal S 
5 whose true nature cannot be determined at the receiver 5, as it may have been 
corrupted by the network 1 3 under test. The network 1 3 imposes degradations on 
the signal S such that the output to the receiver 5 is a degraded version S\ 

This degraded signal S' is analysed by a measurement system 8 to derive an 
objective measure "e" of the errors imposed by the network simulator 3, for example 
10 the bit error rate, packet delay variation ("jitter"), and packet loss characteristics 
(number of packets lost and any pattern to them). It will be appreciated that the 

measure "e" will generally be a vector property (ea, eb, e c , ) as it depends on a 

plurality of independant variables, namely the magnitudes of the various objective 
error conditions such as bit error rate, bit loss rate, and so on). Consequently the 
15 values of M(e) have to be determined over a multi-dimensional search space. 

This measure "e" is fed to an output device which retrieves from the 
calibration store 7 the corresponding value M(e) for output as the subjective 
degradation. Depending on the form in which the data is stored, this may require 
retreival of an entry from a "look-up" table or calculation of a value M(e) for given 
20 values of "e". 

The part of the transmission path upstream of the point in the network 13 or 
the simulator 3 where the known faults are injected (namely transmitter 2) may also 
cause errors in an end to end signal, which may interact with errors imposed by the 
network 3,13, either to make the subjective error worse or to mask it. Any effects 

25 caused by elements downstream of the measurement point in the network 13 may 
also affect the subjective error perceived by the end user, for example if error 
correction software is provided. The process described above only identifies the 
effects of errors introduced by the network 3,13, between the point where simulated 
faults are injected in the network simulator and the point in the real network where 

30 the objective measurement 8 are taken. However, calibration will be effective if the 
end-to-end systems are identical before and after these two points, and the network 
simulator introduces no errors of its own other than those generated by the fault 
simulator. 
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For example, any degradation caused by the transmitter 2 will affect the 
score M generated by the analysis unit 6, as will any degradation or improvement 
(using error correction software) caused by the receiver. Thus the score M is a 
measure of the subjective effect on the signal transmitted through the whole system 
5 2,3,5 (2,13,5) as a function only of those errors imposed by the network 3,13. 
However, provided the transmitter 2 and receiver 5 used in the real test are similar to 
those 2,5 used in the calibration process the subjective effect will be accurately 
modelled. 

In a typical application, the transmitter 2 is a cellular telephone, the receiver 
10 5 is part of the fixed part of the cellular system and its connection to the other party 
to a call, and the system under test 13 is the connection, including the radio link, by 
which the transmitter 2 and fixed part 5 communicate with each other. For any such 
system the nature of the transmitter 2 can be known (e.g. make and model of cellular 
telephone), as can the identity of the receiver system 5. A set of calibration values 
15 M(e) appropriate to this combination can be retrieved for use when this combination 
is in use. If a mobile telephone of a different type makes contact with the network 
1 3, or the original mobile telephone moves to a different base station 5, a different 
set of calibration values generated using this new combination may be retrieved from 
the store 7. 

20 If, for a series of values of "e", the value of M(e) does not vary by much, the 

system may elect to increase the incremetal value between each value of "e". 
Conversely, if M(e) varies by a great amount between each value of ^e" the system 
may reduce the incremental value, or add data for values of "e" interpolated between 
those already taken. This allows the measurements to be concentrated on the areas 

25 of the data space of the greatest interest. 
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CLAIMS 

1 . A method of measuring the perceptibility of degradations caused to signals 

transmitted over a transmission medium comprising the steps of; 
5 generating a test signal; 

generating one or more predetermined transmission degradation conditions; 
subjecting the test signal to the or each transmission degradation condition; 
using an objective measure to measure the degree to which the or each 
transmission degradation condition is perceptible to the human perceptual system; 

generating therefrom a data set for converting transmission degradation 
conditions into values of perceptibility; and 
storing the data set so generated. 

2. A method according to claim 1, comprising the further steps of: 

receiving a signal from a transmission medium; 

identifying transmission degradation conditions in the received signal; 
retrieving from the stored data set a value of perceptibility associated with 
the transmission degradation condition so identified; 

thereby generating a x qrieasure of the perceptibility of the degradation in the 
received signal. 



15 



20 



3. A method according to any preceding claim wherein the transmission 

degradation conditions include values of bit error rate. 

25 4. A method according to any preceding claim wherein the transmission 

degradation conditions include values of packet delay variation. 

5. A method according to any preceding claim wherein the transmission 
degradation conditions include packet loss characteristics. 

30 

6. A method according to any preceding claim wherein values of perceptibility 
are selected according to the nature of elements of the transmission path between 
which the transmission medium under test is interposed. 
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7. A method according to any preceding claim in which an initial plurality of 
degradation conditions are imposed on the system, and further degradation 
conditions are then selected according to the degree of variation in the resulting 
perceptibility measurement values. 

5 

8. Apparatus for measuring the perceptibility of degradations caused to signals 
transmitted over a transmission medium comprising; 

test signal generation means (1) for generating a test signal (St); 
fault simulation means (4) for generating one or more predetermined 
10 transmission degradation conditions (et); 

network simulation means (3) for subjecting the test signal (St) to the or each 
transmission degradation condition (et); 

objective analysis means (6) for measuring the degree (M(e)) to which the or 
each transmission degradation condition (et) is perceptible to the human perceptual 
1 5 system; 

means for generating a data set for converting one or more transmission 
degradation conditions (et) to respective values of perceptibility (M), 
data storage means (7) for storing the data set so generated. 

20 9. Apparatus according to claim 7 further comprising: 

measurement means (8) for receiving a signal (S') from a transmission 
medium (13) and identifying transmission degradation conditions (e) in the received 
signal (S'); 

retreival means (9) for retrieving from the data set in the data storage means 
25 (7) the a value of perceptibility (M) associated with the transmission degradation 
conditions (e) so identified. 



30 
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